Information-theoretic limits of Bayesian network structure learning

While there has been a lot of prior work on determining the information-theoretic limits of structure recovery in Markov random fields (MRFs), which are undirected graphical models, characterizing the information-theoretic limits of learning BNs (directed models) is important in its own right for the following reasons. First, unlike MRFs where the undirected graph corresponding to a dependence structure is uniquely determined, multiple DAG structures can encode the same dependence structure in BNs. Therefore, one has to reason about Markov equivalent DAG structures in order to characterize the information-theoretic limits of structure recovery in BNs. Second, the complexity of learning MRFs is characterized in terms of parameters of the joint distribution over nodes, which in turn relates to the overall graph structure, while the complexity of learning BNs is characterized by parameters of local conditional distributions of the nodes. The latter presents a technical challenge, as shown in the paper, when the marginal or joint distribution of the nodes in a BN do not have a closed form solution. A recurring theme in the available literature on information-theoretic limits of learning MRFs, is to construct ensembles of MRFs that are hard to learn and then use the Fano’s inequality to lower bound the estimation error by treating the inference procedure as a communication channel. Santhanam and Wainwright [13] obtained necessary and sufficient conditions for learning pairwise binary MRFs. The necessary and sufficient conditions on the number of samples scaled as O k2 logm and O k3 logm respectively, where k is the maximum node degree. Information theoretic limits of learning Gaussian MRFs was studied by Wang et al. [14] and for walk-summable Gaussian networks, by Anandkumar et al. [17]. In [18], Anandkumar et al. obtain a necessary condition of ⌦(c logm) for structure learning of Erdős-Rényi random Ising models, where c is the average node degree.